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« The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )|3 Responsive to communication(s) filed on 05 April 2004 . 
2a)M This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) I3 Claim(s) 1-4.8-10,18-23,27-29 and 37-39 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-4.8-10,18-23.27-29 and 37-39 is/are rejected. s 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) 13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 
a)IEI All b)D Some * c)D None of: 

1 .(3 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

A request for continued examination under 37 CFR 1. 1 14, including the 
fee set forth in 37 CFR 1. 17(e), was filed in this application after final rejection. 
Since this application is eligible for continued examination under 37 CFR 
1.114, and the fee set forth in 37 CFR 1. 17(e) has been timely paid, the finality 
of the previous Office action has been withdrawn pursuant to 37 CFR 1. 1 14. 
Applicant's submission filed on April 5, 2004 has been entered. Claims 1-4, 8- 
10, 18-23, 27-29 and 37-39 are pending. All claims except claim 27 have been 
amended. 

It is noted that the amendment filed April 5, 2004 is technically improper 
in that it shows amendments to the claims relative to the proposed After Final 
amendments of February 25, 2002 which were NOT ENTERED. For example, 
the After Final amendment proposed insertion of the language "location" to line 
22 of claim 1 and insertion of "that is represented by the geometric 
information" to line 23 of claim 1. As outlined in the Advisory Action of March 
3, 2004, these proposals did not overcome the rejections of record and were not 
entered. The submission of April 5, 2004 includes this language as now being 
"deleted" from the claim when in fact, it was never entered into the claim. 
However, as the examiner has no difficulty in ascertaining the current claim 
language the April 5, 2004 submission is accepted and an action on the merits 
follows hereinbelow. The notation is made for clarity of the record. 
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Specification 

The abstract of the disclosure is objected to because it is too long, 
containing more that 150 words, the current standard provided by rule. 
Correction is required. See MPEP § 608.01(b). 

Claim Rejections - 35 USC §102 

The following is a quotation of the appropriate paragraphs of 35 
U.S.C. 102 that form the basis for the rejections under this section made in 
this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or 
in public use or on sale in this country, more than one year prior to the date of application for 
patent in the United States. 

Claims 1-4, 8-10, 18-23, 27-29 and 37 - 39 are rejected under 35 
U.S.C. 102(b) as being anticipated by Jarvik (of record). 

The rejection of the prior office actions is maintained and repeated 
hereinbelow. The Jarvik system integrates virtual reality with real-time sensed 
physical reality to provide a unique hybrid environment as is claimed in the 



instant claims. More specifically, the examiner offers the following. 



1. (Currently Amended) A simulator apparatus with which 
an operator plays a simulation in mixed reality space 
including a virtual space and real space in which a real 
object(s] is placed, said simulator comprising: 


Jarvik: virtual reality exercise machine and computer 
controlled video system 


A viewpoint detection unit adapted to detect the 
position/ orientation of a viewpoint of the operator; 


Helmet position sensors (59); utilized in Fig 10 element 
83/ corresponding description 


An inputting unit adapted to input a real space image 
corresponding to the position/ orientation of a the 
viewpoint of the operator; 


Fig 10 element 85/ corresponding description 


A geometric information acquisition unit adapted to 
acquire geometric information of the real object(s) placed 
in the real space; 


Fig 10 element 81/ corresponding description 


A rule memory adapted to store rules for controlling 


Fig 9 element 53/ corresponding description 
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the action of virtual object(s); 




A computation unit adapted to determine the next 
action of the virtual object(s) by referring to said rule 
memory based on a relation among the 
position/ orientation of the viewpoint of the operator, 
position(s) of the virtual object(s) and the geometric 
information of the real object(s); and 


Figs 9, 10, CPU 51 and elements 91, 93, 95, 97/ 
corresponding description 


A virtual space generation unit adapted to generate a 
virtual space image on the basis of the 
position/ orientation of the virtual object(s) after the 
determined action and the position/ orientation of the 
viewpoint of the operator; and 


Fig 10 element 113/ corresponding description 


A mixed reality image generation unit adapted to 
generate a mixed reality space image by superimposing or 
overlaying the virtual space image on the real space 
image. 


Fig 10 element 115/ corresponding description 


2. (Currently Amended) The apparatus according to claim 
1, wherein said inputting unit captures real space images 
of said operator's view of the real space, 


Figs 2, 3, col 7:65-66 at least 


and the apparatus further comprises a video see-through 
type display that the operator wears wherein said the 
mixed reality images are displayed. 


Partially transparent screen 10 


3. (Currently Amended) The apparatus according to claim 
1, further comprising an optical see-through type display 
that the operator wears wherein said virtual space image 
is displayed. 


Helmet 8/ screen 10 


4. (Currently Amended) The apparatus according to claim 
1 , further comprising: a status detector that detects a 
status of the operator, 


Handle forces detected, element 89 


wherein said computation unit determines a next action 
of the virtual object in accordance with the rule stored in 
said rule memory and in correspondence with the 
position/ orientation of the real object and/or the status of 
the operator, and 

computes a position/ orientation of the virtual object after 
the determined action. 


See Figure 10/ corresponding description 


8. (Currently Amended) The apparatus according to claim 
1, wherein the real object(s) include other operators who 
operate said simulator apparatus, and the other operators 
share a single mixed reality space with the operator. 


Col 13:60 -Col 14:5 


9. (Currently Amended) The apparatus according to claim 
1, wherein the real object is an object which is fixed in 
position in the real space, and said geometric information 
acquisition unit comprises: 


Real object = handle 


a predetermined memory for pre- storing ' position 
information and shape information of the real object; 


Sensors 55; stored in memory 


and a reading unit that reads out the position information 
and shape information of the real object from said 
predetermined memory as needed. 


Sensing and computation, element 8 1 


10. (Currently Amended) The apparatus according to 
claim 1 , wherein the real object is an object which is 
movable but does not deform, and 


Real object = handle which is moveable 


Said geometric information acquisition unit comprises: a 
predetermined memory for p re-storing shape information 
of the real object; 


Memory 53 


a position/ orientation sensor for detecting a 
position/ orientation of the real object; and 


Sensors 55 


a setting unit that sets a region the real object is expected 
to occupy in the mixed reality space in accordance with 
the detected position/ orientation of the real object. 


Element 85 


18. (Currently Amended) The apparatus according to 
claim 1, wherein said viewpoint detection unit detects a 
position/ orientation of the head of the operator, and said 
apparatus further comprises: 


Sensors 59 
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a detector that detects a position/ orientation of a hand of 
the operator; and 


Sensors 55 


a recognition unit adapted to recognize a relative position 
of the hand of the operator with respect to the head as a 
command on the basis of an output from said detector. 


Elements 81 and 83 


19. (Currently Amended) The apparatus according to claim 
1, wherein said virtual space generation unit comprises: 
an alignment unit that aligns the position/ orientation of 
the real object to the ' position/ orientation of the virtual 
object after movement; 


Element 99 


a generation unit that generates an image of the virtual 
object after alignment in correspondence with an 
occlusion relationship; and 


Element 1 13 


a head -mounted display device. 


Helmet 8 



A similar analysis can be gleened from the Jarvik reference and applied 
to the remaining claims as they merely reflect the system in terms of a method 
and in terms of a computer program product for implementing the system. Use 
of Jarvik anticipates the method and as Jarvik is computer based, he 
inherently and necessarily includes programming for operability. 

Applicant's current amendments relate to a change in adjective 
descriptors of the various means defined in the claims. For example, 
"location/ posture" has been changed to "position/ orientation" throughout. 
These changes fail to distinguish over Jarvik which as outlined above contains 
the claimed structure/ functions. 

Response to Arguments 

Applicant's arguments filed April 5, 2004 have been fully considered but 
they are not persuasive. Applicant submits that Jarvik fails to disclose or 
suggest determining the next action based upon the interrelationship among 
three factors, namely the position /orientation of the head of the user, the 
position of the virtual object and the geometric information of the real object. 
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In response, the examiner again notes Jarvik Figure 10 and corresponding 
discussion. Jarvik' s computational unit is equivalent to that claimed. In these 
steps, at least the rules of physics are employed to compute where the virtual 
object would appear in .03 seconds, were that object real. Then the object is 
displayed, superimposed with the real objects (position of the handle) in step 
1 15. The data sensed from the real objects (handle - steps 81, 89) are used in 
this calculation, as is the position/ orientation of the user's head (step 83, at 
least). These steps clearly anticipate the now claimed computational unit 
adapted to determine the next action of the virtual objects by referring to said 
rule memory based on a relation among the position/orientation of the 
viewpoint of the operator, position(s) of the virtual object(s) and the geometric 
information of the real object(s). 

Conclusion 

All claims are drawn to the same invention claimed in the application 
prior to the entry of the submission under 37 CFR 1.114 and could have been 
finally rejected on the grounds and art of record in the next Office action if they 
had been entered in the application prior to entry under 37 CFR 1.114. 
Accordingly, THIS ACTION IS MADE FINAL even though it is a first action 
after the filing of a request for continued examination and the submission 
under 37 CFR 1.114. See MPEP § 706.07(b). Applicant is reminded of the 
extension of time policy as set forth in 37 CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first reply 
is filed within TWO MONTHS of the mailing date of this final action and the 
advisory action is not mailed until after the end of the THREE-MONTH 
shortened statutory period, then the shortened statutory period will expire on 
the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In 
no event, however, will the statutory period for reply expire later than SIX 
MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications 
from the examiner should be directed to Jessica J. Harrison whose telephone 
number is 703-308-2217. The examiner can normally be reached on M-F 
during business hours. 

The fax phone number for the organization where this application or 
proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see 
http://pair-direct.uspto.gov. Should you have questions on access to the 
Private PAIR system, contact the Electronic Business Center (EBC) at 866-217- 
9197 (toll-free). A 




Tj&ssicaJ. Harrison 
Primary Examiner 
Art Unit 3714 



